This invention relates to a voice scrambling apparatus for keeping speech communication secret and, more particularly, to a voice scrambler using a frequency-inversion and band-splitting technique. Among conventional voice scramblers, the spectrum inversion scrambler which performs the inversion and rolling of a voice spectrum within a certain frequency band, is most widely used since it can achieve such voice scrambling with high voice quality and less delay time. Examples of such a voice scrambler are disclosed in U.S. Pat. Nos. 4,068,094 and 4,278,840. However, each of these conventional scramblers unavoidably requires a considerable number of types of filters for realizing the spectrum inversion as described hereinbelow.